Papers with MS COCO
ALOHa: A New Measure for Hallucination in Captioning Models (2024.naacl-short)
Copied to clipboard
Suzanne Petryk, David Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph Gonzalez, Trevor Darrell
| Challenge: | Existing metric for object hallucination, CHAIR, is limited to MS COCO objects and synonyms. |
| Approach: | They propose a new open-vocabulary metric, ALOHa, which leverages large language models to measure object hallucinations. |
| Outcome: | The proposed metric correctly identifies 13.6% more hallucinated objects than CHAIR on HAT and 30.8% more on nocaps. |
Learning Relation Alignment for Calibrated Cross-modal Retrieval (2021.acl-long)
Copied to clipboard
| Challenge: | despite advances in multimodal pre-training, cross-modal retrieval remains challenging . lack of relation consistency impairs contextualized representation of image-text pairs . |
| Approach: | They propose a new metric to quantify the relation consistency by measuring the semantic distance between linguistic and visual relations. |
| Outcome: | The proposed method boosts the performance of prevailing models on Flickr30k and MS COCO datasets by a considerable margin. |
Adaptive Weighted Proxy Tuning: Efficient Gray-Box Steering for Image Captioning. (2026.acl-industry)
Copied to clipboard
| Challenge: | Proxy tuning is a decoding-time approach that fails to account for instance-specific variations in model certainty and domain shift. |
| Approach: | They propose a gray-box steering framework that dynamically modulates the logit contributions of a large base model, a fine-tuned expert, and an untune . |
| Outcome: | Adaptive Weighted Proxy Tuning achieves performance parity with fine-tuned models while remaining parameter-free. |
Measuring the Diversity of Automatic Image Descriptions (C18-1)
Copied to clipboard
| Challenge: | a lack of diversity in automatic image description systems is a general problem in natural language generation . authors use established metrics to evaluate system performance on the head of the vocabulary . automatic image descriptions are difficult because of the unbounded range of variation in natural languages . |
| Approach: | They propose to frame automatic image description as a word recall task to quantify the production of generic sentences as 'undiversity' they propose to use established metrics to evaluate the diversity of the output . |
| Outcome: | The proposed metrics evaluate the diversity of sentences generated by state-of-the-art systems on a MS COCO dataset. |
Bridge the Gap: High-level Semantic Planning for Image Captioning (2020.coling-main)
Copied to clipboard
| Challenge: | Recent image captioning models have improved the multi-modal interaction, such as attention mechanisms. |
| Approach: | They propose a high-level semantic planning mechanism that integrates a semantic reconstruction and an explicit order planning mechanism to bridge the gap between visual and language domains. |
| Outcome: | The proposed model outperforms previous methods and achieves the state-of-the-art performance on MS COCO. |
LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation? (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing models for text-to-image generation have been underperforming in image-totext generation tasks. |
| Approach: | They propose a framework that uses a split BERT to create a dedicated latent space for captions and integrates a regularization module to manage varying text lengths. |
| Outcome: | The proposed framework achieves state-of-the-art performance on the MS COCO dataset with 38.2 BLEU@4 and 126.2 CIDEr . |
CapEEN: Image Captioning with Early Exits and Knowledge Distillation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Early Exit (EE) strategies can be used to enhance their efficiency, but their adaptation presents challenges in image captioning as it requires varying levels of semantic information for accurate predictions. |
| Approach: | They propose a framework to improve the performance of EE strategies by knowledge distillation . they use a variant A-CapEEN to adapt thresholds on the fly to account for drifts . |
| Outcome: | The proposed framework gains speedup of 1.77 while maintaining competitive performance compared to the final layer. |
Bridging by Word: Image Grounded Vocabulary Construction for Visual Captioning (P19-1)
Copied to clipboard
| Challenge: | Existing research on image captioning generates frequent n-grams with irrelevant words. |
| Approach: | They propose to construct an image-grounded vocabulary incorporating visual information and relations among words into the decoding process directly. |
| Outcome: | The proposed framework is compared with state-of-the-art models on MS COCO and Flickr30k and shows that it is more efficient than existing models. |